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l.f  ! . '*  / F-V.  1 c rf  1 a t iopsh  i p'  of  reac'a*  1 1 i ‘ v (as  r *\->  surer  )■■«»  t>e 
Klesch  ana  Kincaid  formulas)  and  conceptual  level  (ri.)  to  r-orforpance 
or  study  nun^t ions  contained  in  the  rion-her  jdenh  fsrprr  ronrse  (nT'CC) 
Hull  - V.intenaiice  Technician  3i»2,  fJAVTDTRA  °1510  were  aralvzed.  Two 
samples  of  studv  questions,  a nonerror  sarnnle  and  an  error  sar  nie, 
vcre  Grav -n  from  this  NRCC.  r"he  results  sho’-'  that  ’■padaHlit.v  bet’-’eor 
the  nonerror  and  the  error  samples,  or  between  the  textual  oassaoes 
associated  v it),  these  samples,  does  not  differ  siqnif icantly . The 
results  do  show,  however,  that  the  concentual  level  (thinking)  reouired 
to  answer  the  study  questions  is  related  to  error  rate.  That  is,  hicib 
error  rate  study  ouestions  are  associated  with  hioher  conceptual  levels 
than  are  zero  error  rate  study  questions.  Methods  are  presented  for 
using  high  and  low  CL  study  questions  to  improve  the  traininq  effective- 
ness of  MPCCs.  The  recommendation  is  nade  that  retention  of  hiqh  and 
low  conceptual  level  study  questions  he  analyzed  because  previous 
research  has  indicated  that  better  learnino  and  memory  may  he  associated 
with  study  questions  that  are  highly  conceptual,  "he  recommendation  is 
also  made  that  the  relationship  between  NPCC  performance  and  verHil 
intell  ioonce  (measured  tftrouqh  the  General  Classif  ication  ^est)  be 
determined  because  verbal  intelligence  is  most  likely  involved  in 
performing  well  on  highly  conceptual  study  questions.  In  addition, 
readability  formulas  should  be  developed  to  include  conceptual  level  as 
a factor  in  determining  readinq  orade  levels.  The  inclusion  of  a 
conceptual  level  factor  in  readability  formulas  could  improve  the 
validjtv  of  these  formulas. 
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3.0  PimPOfT . This  report  presents  a preliminary  analysis  of  the 
relationship  between  performance  on  study  questions  which  were 
administered  routinely  in  conjunction  with  a current  Mavy  Mon -Resident 
Career  Course  (MP.CC)  and  the  readability  of  the  Pate  Training  Manual 
(RTF)  passages  from  which  these  study  questions  were  tal  en.  In 
addition,  the  association  betveen  study  question  performance  and  the 
readability  of  the  study  questions  will  be  determined , as  well  as 
the  relationship  betveen  study  question  performance  and  the  conceptual 
level  renuired  to  answer  the  questions  correctly  usinq  the  information 
provided  in  the  associated  r.T'i  passage.  The  data  and  interpretations 
found  in  this  report  nay  be  useful  to  the  Chief  of  'laval  1'ducation  and 
Training  6upport  and  others  within  and  outside  the  Maval  Fducation  and 
Training  Conjiiand  in  developing  and  implementing  policies  and  procedures 
to  make  readinq  materials  and  nonresidential  instruction  more  useful 
to  the  consumer. 

4.C  EACKGROrND . A number  of  recent  reports  (references  1,  2,  3,  and 
4)  have  demonstrated  that  a substantial  difference  mav  exist  between 
the  readability  of  Navy  written  materials  and  the  reading  skills  of  the 
bavy  personnel  who  are  required  to  read  these  materials  in  order  to  per- 
form well  or  to  advance  in  rate.  Readability  of  written  materials  is 
measured  by  a variety  of  formulas  which  are  highly  intercorrelated  and 
which  are  purported  to  measure  the  level  of  reading  difficulty  in  terms 
of  the  average  grade  level  of  education  which  would  be  required  to 
comprehend  the  textual  information.  This  average  education  level  is 
referred  to  as  lead  inn  Grade  Level  (RGL) . 


3 


In  rest'onse  to  the  alovc  findinas*  the  reconmendation  has  hern 
niacin  (OP-099  seeedletter  991I./550  of  13  /’arch  1974)  that  the  PfIT.s 
of  havy  trainin'?  am.  technical  materials  should  be  reduced  in  order 
to  improve  occupational  performance  and  advancement  opportuniti es  arono 
/’aval  personnel.  Later  recommendations  (reference  1)  cautioned  aaairst 
arbitrary  lowering  of  PGLs,  and  stated  that  while  the  formulas  which 
are  used  to  measure  POLs  mav  he  useful  in  monitorino  the  overall  readira 
difficulty  of  written  materials,  the  validity  of  these  formulas  in 
measurin'?  accurately  the  readinq  comprehension  of  Navv  enlisted  personnel 
for  ’law  technical  and  traininq  materials  has  not  been  established 
satisfactorily.  (Pn  extensive  discussion  of  the  validity  problems 
associated  with  these  formulas  is  provided  in  reference  5.)  Inasmuch 
as  most  of  these  formulas  determine  RfJLs  usinq  some  combination  of 
averaqe  sentence  lenqth,  number  of  syllables  per  word,  ar.d  the  frequency 
with  V'hich  words  are  used  in  common  forms  of  i/riting,  the  possibility 
exists  that  writers  could  use  shorter  sentences,  reduce  the  complexity 
of  words,  and  use  more  frequent  or  common  words  and  still  write  material 
which  is  difficult  to  comprehend.  In  addition,  little  has  been  done 
either  amonq  military  or  civilian  organizations  to  show  that  conscientious 
use  of  these  parameters  (shorter  sentences,  less  complex  words,  and 
more  common  words)  results  in  improved  comprehension  or  performance. 
Although  the  Pir  Force  (reference  6)  has  demonstrated  that  reductions 
in  the  Automated  roadability  Index  (ARI)  for  a technical  manual 
significantly  improved  performance  on  an  associated  multiple-choice 
test,  these  findings  must  be  interpreted  with  caution.  The  absence  of 
pre-  and  post-revision  samples,  and  the  failure  to  lave  experienced 
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editorial  writers  compare  content  ant!  rtvlc  char act eristics  Lrtwon 
the  two  manuals,  leave  open  the  possibility  that  modifications  in  con- 
tent and  style  (in  addition  to  reduced  f.GLs)  m ay  account  for  the 
nerforrancc  improver  onts  which  were  cA  served . The  advice  found  in 
professional  writing  guides  (references  7 and  R) , as  veil  as  t^e 
roconr/rndations  nadc  by  researchers  (references  1 and  0)  , would 
indicate  that  ’Gbs,  while  important  to  total  readinq  comprehension, 
sIkjuIU  not  be  emphasized  to  the  exclusion  of  content  and  style. 

7 nroSler  that  van  ignored  in  the  nlove  fir  Torce  research,  and 
tliat.  is  often  ignored  Ly  writers  as  well  as  those  who  are  responsible 
for  using  written  materials  in  training,  is  the  relationship  between 
test  questions  and  the  textual  material  which  provides  the  information 
necessary  to  answer  these  questions.  Little  is  known  about  the 
relationship  of  item  error  rate  to  (a)  the  readability  (RGLs)  of  the 
questions,  and  (b)  the  extent  to  which  questions  duplicate  the  textual 
material  (i.  e.,  differences  in  error  rates  between  verbatim  questions 
and  those  questions  which  require  the  reader  to  conceptually  orqanize 
or  think  about  the  written  information  in  order  to  provide  a correct 
answer) . The  possibility  exists  that  comprehension,  as  measured  by 
test  questions,  is  not  related  solely  to  the  readabilitv  and  style  of 
the  textual  raterial,  but  is  related  as  well  to  the  readabilitv  of  the 
test  questions  and  to  the  conceptual  level  required  to  answer  the 
questions  correctly.  Previous  findings  (references  R and  10)  have 
indicator,  that  the  study  question  technique  used  in  Navy  NRC C courses, 
in  which  the  test  questions  (referred  to  as  "study  questions”)  are 
used  to  auide  a reader  t.hrouqh  the  RTMs,  nay  not  be  train.inq  effective. 
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This  technique  may  he  ineffective  for  retention  of  the  textual 
information  which  answers  the  study  questions  and  for  learninn 
textual  information  which  is  incidental  to  these  questions.  The 
results  found  in  reference  10  show  further  that  error  rates  amonq  studv 
questions  appear  to  he  related  to  the  extent  to  which  the  reader  rust 
(a)  search  for  the  answer  in  the  RTM,  and  (h)  irteqrate  several 
different  types  of  information  in  answer ino  the  question,  mhe  findings 
in  reference  9,  however,  may  not  be  valid  for  Maw  use  because  of  the 
snail  sample  size,  the  hiah  readinq  skill  level  of  the  subjects,  and 
inadequate  research  design  and  statistics,  while  reference  10  used 
hiqhly  subjective  techniques  to  judge  the  textual  factors  that  wore 
related  to  error  rates  amonq  the  studv  questions. 

The  following  data  were  collected  in  order  to  determine  if  error 
rates  for  NRCC  studv  questions  are  related  siqnif icantlv  to  (a)  the 
readability  (Pdl.s)  of  the  textual  materials  (RTMs)  which  contain  the 
answers  to  these  questions,  (b)  the  RGLs  of  the  study  questions,  and 
(c)  the  extent  to  which  the  RTM  information  must  be  integrated  and 
organized  conceptually  by  the  reader  in  order  to  answer  the  studv 
questions. 

5.0  METHODS . The  following  subjects,  procedures,  and  statistical 

techniques  were  used  in  collecting  and  analyzing  the  present  data. 

2 

5.1  Subjects.  The  subject  group  consisted  of  30  Navy  personnel 
who  submitted  answer  sheets  in  order  to  obtain  credit  for  the  KPCC 
entitled  "Hull  Maintenance  Technician  3&2"  (IIMT  3k?.)  . The  Jn'*TP  3s 2 
course  was  used  because  (a)  this  course  was  judged  hy  several 
experienced  Navy  writers  to  be  well  written^,  (h)  the  I'm  (Hull 
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Technician)  ratina  was  judged  to  be  a fairly  representative  Navy 

4 

ratine  , and  (c)  much  of  the  data  had  alreadv  been  submitted  and 
analyzed  in  coi' junction  v/it'n  another  NPCC  research,  project. 

The  bin  3r.2  cours-  is  taken  routinely  by  apprentices  (or  ’strikers”) 
who  are  tryinn  to  nualify  for  the  ITT 3 (bull  Technician  Third  class) 
advancer  ert  in  rate  examination,  or  by  those  personnel  who  are  rated 
alreadv  ar  hT3n,  > ut  who  arc  tryino  to  advance  to  IT' 2 (Hull  Technician 
recond  Class).  The  present  sample  is  not  necessarily  r "presentative 
of  tiro  total  IT'  Apprentice  and  1.T3  [ropulations  enrolled  in  the  ITT 
3£2  course,  tut  is  more  likely  to  bo  represontat j we  of  those  personnel 
who  completed  the  course  and  whose  commands  (shins  or  stations)  were 
diligent  and  cooperative  enouqh  to  return  the  completed  answer  sheets. 

The  present  sample  therefore  probably  represents  a more  intelligent,  and 
rrotivateu  group  than  those  who  did  not  complete  the  course  or  who  were 
assigned  to  commands  which  neglected  to  return  answer  sheets.  In 
addition,  the  present  subject  group  was  taken  fron  amona  those  who  first 
submitted  answer  sheets,  so  the  possibility  exists  also  that  thev  may 
not  bo  representative  of  the  total  group  which  eventually  submitted 
these  sheets. 

Although  aptitude  and  career  data  were  not  available  for  the 
present  suvjects,  a review  was  made  of  these  data  for  ITT  Apprentices 
arid  bT3s  who  had  passed  the  I’MT  3&2  course  successfully  and  wlx>  took, 
the  examinations  for  advancement  to  the  next  higher  rate  (IT3  and  HT2) 
in  August  of  1975.  These  data  included  average  GCT,  the  percentage  of 
personnel  at  the  UT  Apprentice  and  IT3  levels  who  had  been  at  these 
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rate  lev*  Is  12  months  or  lens  (nonthn  in  rate),  and  the  percentage 


of  personnel  at  cadi  of  these  rate  levels  who  had  Icon  in  the  h'avy 
3 ( months  or  less  (lenoth  of  service).  Tn  order  to  determine  whether 
or  not  !ri'  is  a representative  f-’avy  rating,  the  al  ove  aptitude  and 
career  data  for  HT  Apprentices  and  HT3s  were  compared  to  similar  data 
for  personnel  taking  any  August  107c  examination  for  advancement  to 
compaiable  rate  levels  (Overall  Apprentice  and  Overall  Petty  Officer 
3 arouns) . 

5.2  Mater ial s . The  PA1T3&2  course,  similar  to  most  MRCCs,  consists 
of  an  assianment  booklet  and  an  PT!!.  Fach  assignment  in  the 
| assignment  hooklet  covers  1 or  more  subject  natter  areas  in  Full 

.'laintenar.ee  Technology.  The  chapters  in  the  PT"  discuss  in  detail 
the  subject  matter  areas  presented  in  the  assignment  booklet.  Pack 

assionrent  consir.tr.  mostly  of  multiple-choice  study  auestions  and  a 

i 

i 

few  true -false  and  matchina  study  questions.  Previous  research  (reference 

1)  has  si  own  that  the  P.TM  for  HTt'3f<2  has  an  overall  Flosch  PCI.  between 

t he  10th  and  12tl  grades  and  a Fincaid  ROL  of  10.72  grades,  "he  hTM 

Jfc?  assign). ent  Iooklet  l.as  an  overall  Flesch  SOI.  between  the  10th-12th 
5 

grades. 

Loti  the  P.TM  and  assignment  booklet  are  issued  together,  and  the 
course  is  completed  at  the  convenience  of  the  student  during  off-duty 
hours  with  minimal  supervision.  The  student  is  told  to  study  the 
specific  pages  of  the  RTM,  which  arc  identified  at  the  beginning  of 
each  assignment,  and  then  answer  the  associated  study  questions  in  the 
assignment  booklet.  As  demonstrated  in  reference  10,  however,  the  reverse 
procedure  appears  to  occur  more  often — the  student  analyzes  the  questions 
and  then  tries  to  find  the  answers  in  the  RTM.  This  procedure  orohablv 
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makes  for  sporadic  and  mixed  coverage  of  the  textual  material  contained 
in  the  RTMs.  Responses  to  the  multiple-choice  study  questions  are 
entered  on  a separate  answer  sheet  by  erasing  1 of  4 columns  (2 
columns  in  the  case  of  true-false  questions;  2,3,  or  4 columns  for 
matching  questions) . If  an  incorrect  answer  is  entered  (erased) , the 
incorrectly  erased  column  refers  the  student  to  the  RTM  page  which 
begins  the  topical  discussion  from  which  the  correct  answer  may  be 
derived.  The  student  then  reads  this  material  and  enters  another 
response.  The  appropriate  RTM  material  is  not  too  difficult  to 
locate  because  the  questions  usually  cover  sequential  RTM  material, 
a condition  which  probably  encourages  the  student  to  refer  to  the 
questions  before  reading  the  text.  Underlying  the  answer  sheet  is 
a duplication  sheet  which  provides  a permanent  record  of  the  erased 
responses  so  that  a determination  can  be  made  of  the  number  of  responses 
entered  before  the  correct  answer  was  chosen. 

For  the  purpose  of  the  present  analysis,  those  study  questions 
which  had  2 or  more  erasures  were  counted  as  incorrect,  while  those 
questions  which  had  a single  erasure  were  counted  as  correct.  Only 
the  study  question!  contained  in  the  first  12  (of  27)  assignments 
were  analyzed  in  order  to  reduce  the  data  collection  and  statistical 
requirements.  In  addition,  only  multiple-choice  and  true-false  study 
questions  (and  the  RTM  passages  associated  with  these  questions)  were 
included  in  the  analysis  because  the  readability  of  matching  questions 
was  too  cumbersome  and  unreliable  to  determine.  In  order  to  further 
maximize  the  differences  between  the  study  questions,  and  to  improve 
the  classification  of  the  study  questions  into  error  and  nonerror  samples, 
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only  those  questions  which  were  not  missed  by  any  of  the  subjects 
(nonerror  sample)  and  those  questions  which  were  missed  by  at  least 
5 of  the  subjects  (error  sample)  were  used  in  the  present  analysis. 

This  differentiation  resulted  in  106  questions  in  the  error  sample  and 
77  questions  in  the  nonerror  sample.  In  order  to  make  reliable  judge- 
ments of  the  reading  grade  levels  (see  section  5.3  below),  those  study 
questions  that  referred  to  diagrams  in  the  RTM  were  also  removed  from 
the  2 samples.  As  a result  of  this  procedure,  7 study  questions  were 
removed  from  the  error  sample  and  3 study  questions  were  removed  from 
the  nonerror  sample.  The  answers  to  the  remaining  questions  in  the 
error  sample  and  the  nonerror  sample  could,  therefore,  be  referenced 
directly  to  a specific  and  delimited  passage  of  textual  material  within 
the  RTM.  These  textual  passages  were  provided  by  the  writers  of  the 
HMT  3 & 2 course . 

In  an  additional  sampling  procedure,  study  questions  were 
removed  from  further  analysis  if  either  the  study  question  or  the 
associated  RTM  passage  did  not  conform  to  the  readability  parameters 
established  by  Flesch  (reference  11). ^ This  procedure  resulted  in 
disregarding  19  questions  from  the  error  sample  and  9 questions  from 
the  nonerror  sample.  Following  this  procedure,  the  total  number  of 
study  questions  in  the  error  sample  was  80,  while  65  study  questions 
remained  in  the  nonerror  sample. 

5. 3 Measuring  RGLs  and  Conceptual  Levels.  The  Reading  Grade  Levels 
( RGLs ) of  the  RTM  passages  described  above,  as  well  as  the  RGLs  of 
the  associated  study  questions,  were  determined  using  both  the  Flesch 
Reading  Ease  Score  (the  Flesch  formula)  and  the  correctioi.  to  the  Flesch 
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forrula  which  vas  developed  recently  ly  Kincaid  (the  Kincaid  formula). 


The  procedure*  involved  in  applying  these  formulas  to  this  tvr>e  of 
'HCC  raterial,  which  arc  too  extensive  to  describe  in  detail  in  this 
report,  have  been  presented  fully  in  an  earlier  CNFTS  publication 
(reference  1).  described  briefly,  however,  the  Flesch  formula  uses 
the  number  of  syllables,  words,  and  sentences  in  textual  samples  of 
at  least  100  words  to  determine  Tin,.  Tn  the  case  of  those  PTN 
massages  and  study  ouestions  which  consisted  of  less  than  100  'words, 
the  ."CLs  were  determined  using  correction  procedures  recommended  by 
Flesch  (reference  11).  The  Kincaid  formula  modifies  the  resultinn 
Flesch  m,  by  correcting  for  the  reading  shills  which  ’'avy  personnel 
lave  ua >onstrated  for  this  type  of  davy  trainir.q  material.  The  Kincaid 
forrula  has  been  found  to  reduce  the  majority  of  Flesch  PCLs  for  KP.CCs 
between  1 and  2 grade  levels  (reference  1). 

The  Conceptual  Level  (Cb)  required  by  the  students  to  answer  the 
study  ouestions  from  the  textual  information  provided  in  the  associated 
I.Td  passages  was  determined  by  2 independent  judaes.  ""he  judqes  were 
psychology  students  (a  female  and  a male)  who  were  seniors  at  a local 
university.  both  judges  were  a'-out  25  years  of  age  and  were  earning 
above  average  college  grades.  Neither  judoe  van  familiar  vith  Tr5CC 
writing  or  testing  procedures,  and  neither  judge  )iad  experience  vith. 

IT""  terminology  or  procedures. 

'i  iie  2 judges  were  as)  ex  I to  use  a 4-point  rating  scale  in 
determining  the  extent  to  which  fIF.CC  students  would  have  to  thinh  about 
the  information  which  was  • roviiied  ir.  the  associated  FT**,  massages  in 
answering  each  of  the  140  total  study  ouestions  (f>0  ouestions  in  the 
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er ror  rar-ple  ard  r *5  questions  ir  the  nonorror  sample).  m]1f,  d-.-x-iinf 
rating  scale  was  ada?  ted  fron  the  first-  A.  mnnit-ivn  domain  levels 
developed  Y y hlooi.  (reference  12).  The  -judges  first  read  the  question 
stem  and  the  4 choices  (or,  in  the  case  of  true-false  iters,  the  iter 
statement)  , an<.  then  read  the  associated  ::T”  passage  before  ra'ino 
each  iudur.cnt . They  were  all oved  to  re-read  th.e  passage,  statement, 
question  stem.,  and  cloices  as.  often  as  thev  wanted  before  mab’nq 
a judgement.  They  were  not  told,  however,  which  choice  was  correct 
before  ratine  tbe  studv  question.  The  ratinq  scale,  rawinn  fror.  a 
score  of  1 (lowest  Cl. --little  thinkinq  reouired  to  answer  the  studv 
euestion)  to  4 (biqhest  CL — much  thinkinq  and  organization  required  to 
answer  the  study  question) , is  described  in  Appendix  t . 

The  4-point  scale  was  explained  carefully  to  the  -judges  at  the 
start  of  a training  session,  and  they  were  then  allowed  to  practice 
on  study  auestions  taken  fror  later  T!Tf*3f.2  assiorrents  (assignments 

13  to  27) . These  practice  study  questions  consisted  of  an  equal  nurber 
of  error  study  questions  (tl->ose  questions  nissed  by  at  least  5 of  the 
subjects)  and  nonerror  studv  questions  (those  ouestions  which  were  not 
missed  by  any  subject) . The  error  and  nonerror  questions  were  arranged 
in  random  order.  After  ratinq  each  practice  question,  the  judaes 
explained  the  loqic  they  used,  in  making  these  ratings.  'T'hev  continued 
practicing  until  they  demonstrated  consistency  in  separately  scoring 

14  out  of  15  practice  questions  (this  criterion  was  reached  after 

2 practice  trials) . The  2 judqes  then  rated  the  145  studv  questions 
for  3 hours  a day  over  the  next  5 days,  The  judqes  also  scored  5 
practice  auestions  at  the  beginning  of  each  of  these  5 days  in  order 
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to  maintain  familiarity  with  the  ratine;  procedures . The  j uan or,  wrrf 
free  to  use  a dictionary  while  rea.liro  the  T'TV  passages  and  studv 
questions,  and  to  consult  with  nfTT  writers  and  sut  ject  natter  pyn»rts 
during  this  period  (neither  of  which  thev  did).  The  judges  c'id  not 
communicate  with  each  other  while  ra! ire  tf,e  iudoeronts,  ror  wore 
they  told  which  stuuy  oucstions  tnro  in  the  error  and  ncrerror  sarnie®. 

/ fter  rating  each  stutly  cuestion,  the  judoes  v-ore  told  the  correct  choice, 
as  well  as  the  ratings  they  each  liad  made  of  that  study  question . If 
these  ratings  differed,  they  were  ashed  to  explain  the  logic  which  they 
had  use;1  in  rial  ing  the  iudaenents.  The  original  judgements,  however, 
were  used  for  calculating  the  present  results  and  were  rot  modified 
after  these  explanations. 

5.4  rtatistical  Irocedures.  Reliability  between  the  2 judoes  in  ratine 
CL.s  was  determined  using  an  intraclass  correlation.  In  order  to  insure 
that  these  separate  fl,  judgements  not  only  varieti  in  a consistent  fashion, 
but  were  truly  similar,  a t-test  for  indeoenrdent  samples  was  use*1  to 
compare  the  means  of  the  ratings  made  hy  each  judge.  Oifferonces  ir 
rots  and  rI.r,  for  t!ic  error  and  nonrrror  samples  ierr  also  tested  for 
significance  using  t-tests  for  independent  samples.  (The  CT  score  for 
each  study  cuestion  wifhir  t)ie  error  and  nonerror  samples  was  tlie  average 
of  the  ratines  a-sinred  to  that  question  hy  the  ">  iudde®.)  "lo  t-te*-*- 
lor  inac;  tni.cnt  ®arples  vis  used  a;j  w».'ll  for  cor  paring  the  r,C"  scores  of 
I1T  Apprentices  and  PT3s  with  Overall  /j prentice  and  Tetty  Officer  3 nroups, 
wl.ilo  the  chi-square  test  was  used  to  compare  differences  in  career  d;tta 
(length  of  service  and  months  in  rate)an©no  these  groups . For  the  <~.rr 
data,  variances  were  Courd  to  differ  significantly  for  the  i’T  ’ nprcnticc 
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and  PY3  nroim:,  on  comparison  to  the  Overall  Apprentice  arc1  poi  groups. 
Consoouentlv,  the  rochran  and  ro:c  method  (reference  11)  v.-as  used  to 
compute  the  t tests  involving  GC"  Means.  bevels  of  riqri f iear.co 
for  all  tests  were  p£.  0r>  (tvr>- tailed)  . 

(.'  T'!fTLTT.  The  followirq  sections  present  the  f inclines  for  cor- 
l-arisons  Letween  liTs  and  other  ratinqs,  differences  in  ff’.r.  betveer 
the  error  and  nonerror  sar.nl  or, , reliability  of  the  juoqcs  in  Pet  err  i r iro 
th.e  CLs  of  the  r.tudv  questions,  anc!  differences  in  rT,s  between  the  error 
and  nonerror  sar.ples. 

C.l  fon.nar  i sons between  ITs  and  Other  ratinos.  ’T'a*de  ' present;  t he 

GCT  and  career  data  for  PI  Apprentices  and  PT3s,  as  well  as  PC'"  arc' 
career  data  for  those  who  were  in  the  Overall  Apprentice  and  Petty 
Officer  3 qroupr..  This  table  shows  that  sinni  f icar.tly  roro  >'orsorre] 
had  leen  at  the  !.T  Apprentice  and  IT3  rate  levels  13  months  or  less 
than  personnel  at  comparal  le  rate  levels  in  the  Overall  Apprentice 
and  Petty  Officer  3 groups.  In  addition,  siqnif icantlv  more  p^ls  had 
Leen  in  the  Navy  3f>  ronths  or  less  than  personnel  in  the  Overall  Pettv 
Officer  3 qroup.  The  proportion  of  IT  Apprentices  and  Overall  Apprentices 
who  had  been  in  the  Navy  3f>  months  or  less  did  not  differ  siqnif  icantlv. 
Table  1 also  provides  evidence  that  both  IT  qroups  were  sionif icantlv 
lower  in  verbal  irtell joence  (got  scores)  than  the  overall  Apprentice 
and  Petty  Officer  3 groups.  Inasmuch  as  these  data  show  that  ’T  Appren- 
tices and  IT3s  are  not  representative  of  other  (overall)  comparable 
Navy  ratinqs  in  verlal  intelliqence  and  months  in  ra'e,  and  that  ’"'’’le 
are  not  representative  of  other  Pettv  officer  3s  in  Ion.-*-'  of  service, 
the  results  described  below  (sections  0.2  and  G.4)  mav  not  be  replicated 
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for  personnel  in  other  Navy  ratings.  A further  assumption,  which 


remains  unsubstantiated,  is  that  the  subjects  used  in  the  present 
analysis  would  be  comparable  to  these  HT  Apprentices  and  HTls  in 
GCT  scores,  months  in  rate,  and  length  of  service. 

6 . 2 Differences  in  RGLs  Between  Error  and  Nonerror  Samples. 

"he  mean  RGLs  resulting  from  the  Flesch  and  Kincaid  formulas  for 
the  study  questions  and  associated  RTM  passages  in  the  error  and 
nonerror  samples  are  shown  in  Table  2.  The  t-tests  presented  in  this 
table  show  that  differences  in  Kincaid  and  Flesch  RGLs  between  the 
2 samples  were  not  significant,  either  for  the  RTM  passages  or  the 
-.ludy  questions.  (A  noteworthy  finding  is  that  RGLs  for  the  error 
sample  were  lower  than  RGLs  for  the  nonerror  sample  in  most  cases, 
although  these  differences  did  not  reach  significance.)  Differences 
in  error  rates  between  the  2 samples  are  unlikely  therefore  to  be 
attributed  to  the  readability  of  either  the  RTM  passages  or  study 
questions.  The  average  RGLs  for  the  RTM  passages  and  study  questions 
in  the  error  and  nonerror  samples  also  correspond  closely  to  the 
average  RGLs  found  for  the  entire  HMT3&2  assignment  booklet  and  RTM 
in  previous  research  (reference  1) . 

6.3  Reliability  of  CL  Judgents.  The  2 judges  were  found  to  be  in  high 
agreement  in  rating  the  CLs  of  the  145  study  questions.  The  intraclass 
reliability  between  the  2 judges  for  the  145  ratings  was  0.836.  The 
total  sample  of  145  study  questions  and  associated  RTM  passages  were 
assigned  an  average  CL  rating  of  2.36  (sd  = 0.91)  by  the  first  judge, 
and  an  average  CL  rating  of  2.32  (sd  = 0.93)  by  the  second  judge.  The 
appropriate  t-test  showed  that  these  means  did  not  differ  significantly 

(t  = 0.369;  df  = 288;  p = ns).  Although  these  results  do  not  substantiate 
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the  val  i«.ity  of  these  judgements , this  level  of  aiTwoni  in  raMr'*-. 
i ndi ca t e r.  that,  under  t'  osr  conditions,  the  " judoor-  wore  Pine]’/ 
con. si.  tort  li.  applying  the  standards  which  wore  used  to  rate  tie  r 
of  these  studv  injt-it ions . Inanples  of  study  nuestiors  and  assoc iate< : 

’'?**  massages  vMcl  were  rated  ’ y the  judges  at  the  * conceptual  Jeer’s 
are  presentee.  in  / ; '['end .it  1 . 

< ' differences  in  fj.s  between  hrror  anf  ‘lorerror  fai’nlon. 

'.lie  mean  ch  score  for  the  error  sample  was  ?.r3  (sd  = o.f'R)  , <.•*- i l e *-»,r 
roan  Cl,  score  for  tie  nonerror  sarple  was  ".11  (:-,t i = n.7<  ) . "Tp 
t-tost  result  inn  from  a comparison  of  these  2 roans  was  f oun<’  to  t p sin- 
i if  i cant  (t  — 3.00;  of  = 1^13;  r,  4.005)  . '"Oese  findings  slow  that  the 
averaqe  (7,  score  for  the  error  sauj-le  was  sionif icantly  higher  than  the 
average  CL  score  for  the  nonerror  sanple.  This  firOinct  indicates  that 
conceptual  difficulty  is  involved;  significantly  in  t'-'e  different,  error 
rates  between  the  2 samples. 

7.0  niFCUSf  IO?T.  "’lie  a!  ove  results  show  that.  reeclal  j 1 ity , as  reaRiirn! 

Ly  the  ! lescli  and  Kincaid  formulas,  has  little  to  do  "ith  performance 
(error  rates)  on  the  bRCC  for  I?''!'3&2  wliich  is  adm  ini  stored,  under  these 
conditions.  These  findings  appear  to  demonstrate  that  stylistic  writing 
factors,  such  an  those  factors  represented  ly  ci.  scores,  nay  he  no  re 
important  than  readability  to  this  t'g~>e  of  performance  in  these  training 
situations,  before  discussing  these  FGI,  and  CI,  effects  further,  however , 
the  present  results  sltould  he  interpreted  with  caution  because  (a)  the 
size  of  the  subject  group  was  small  (’3  = 30)  and  selective  (chosen 
fron  anone  tliose  personnel  who  1-oth  completed  and  returned  the  >'tf”3f? 
course) , (b)  the  alove  OCT  and  career  data  indicate  that  I’T  Apprentices 
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arx.  IT3s  (the  jionulations  from  which  the  present  subjects  are  assumed 


to  ho  drawn)  an  no^  ert.irely  representative  of  other  ’’avy  ratings, 

(c)  the  l.rfT3G?  course  mav  not  to  representative  of  other  ,mCT:s 
inasmuch  as  these  177T3&2  course  materials  arc  recoor.izot'  ly  profess- 
ional 'T-CC  writers  as  heinn  especially  well  written,  (d)  the  validity 
of  CL.  ratine's  hay  vet  to  Le  demonstrated  fullv,  (e)  although  the 
present  findings  indicate  t)iat  CL  ratinqs  may  he  involved  siqnif icantly 
in  h'fCC  performance,  this  conclusion  rust  await  cross-validation  of 
the  present  results  usinq  different  judges,  and  (f)  measures  of  reada- 
bility other  than  the  riesch  and  Kincaid  formulas  (especially  measures 
Which  involve  comprehension  of  this  tvno  of  course  material  a no no 
those  specific  havy  personnel  who  must  read  these  materials  in  order 
to  oualifv  for  rate  advancement)  mav  vet  demonstrate  a relationship 
between  readability  and  !!hCC  performance.  In  addition,  the  importance 
of  conceptual  level  indicates  that  measures  of  verlal  intelligence 
(such  as  cf")  may  he  related  to  H'OC  performance,  with  those  personnel 
who  have  higher  CCT  scores  performing  better  than  those  with  lower 
CCT  scores.  This  effect  ray  be  most  apparent  for  questions  in  the  error 
sample  which  have  high  CT,  scores.  This  effect  could  not  be  analvzed  in 
the  present  situation,  however,  because  CCT  scores  wore  not  read.ilv 
available  for  tins  subject  group.  mv,e  above  precautions  indicate  that 
(a)  the  present  findirqs  should  lie  replicated  on  a much  larger  and  more 
representative  subject  group  (from  a variety  of  T’avy  ratinos  and  rate 
levels),  (b)  additional  KPCCs  should  be  used,  (c)  f'L  ratines  should 
be  replicated  and  validated,  (d)  replication  s!»uld  involve  readability 
measures  other  than  the  riesch  and  Kincaid  formulas,  and  (e)  a determination 
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shoulu  Lc  made  of  the  relationship  between  j>oor  NTcCC  performance 
(hioh  error  rates)  and  verbal  intelligence. 

The  present  findings  appear  to  reemphasize  the  basic  deficioncv 
in  most  readal  ility  formulas— the  failure  to  account  extensively  and 
accurately  for  readinq  comprehension.  rphese  results  sl'ow  that  formulas 
which  manipulate  sentences,  words,  and  syllables,  even  if  corrected  for 
the  readinq  shills  which  ’"avy  personnel  have  demonstrated  for  related 
tutorial,  cannot  measure  the  fine  and  subtle  conceptual  relationships 
which  appear  to  exist  between  textual  material  and  criteria  of  comore- 
hension  and  performance  effectiveness  (such,  as  error  rates  for  nuestions) . 
These  formulas  can  account  only  for  that  qross  level  of  comprehension 
(or  eoncertual)  difficulty  which  is  related  to  sentence,  word,  and 
syllable  complexity,  and  can  do  little  to  differentiate  the  conceptual 
level  required  to  comprehend  material  which  is  similar  in  sentence, 
word,  and  syllable  structure.  The  present  analysis  overcame  this 
deficiency  by  liaving  judqes  who  were  naive  about  the  su)  iect  matter 
(but  who  were  not  necessarily  naive  alxsut  study  question  difficulty) 
rate  the  conceptual  level  which  was  required  to  comprehend  the  textual 
material  (RT*!s)  sufficiently  well  to  perform  effectively  on  the  studv 
questions.  These  results  indicate  that  readability  consists  of  more 
than  sentence  lenqth,  v.r> rd  froouenev,  and  word  complexity,  but  consists 
also  of  the  cognitive  processes  involved  in  organizing  and  internretjnn 
textual  material.  Although  the  readino  shill  which  measures  these 
cognitive  processes  is  referred  to  as  "comprehens ion1' , most  roadinn 
experts  have  failed  to  specify  comprehension  in  term  if  the  relationship 
which  exists  ) etween  textual  material  and  performance  crit  ia,  while 
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none  of  the  available  readine  formulas  incorporates  a separate  compre- 


hension or  CT.  factor  into  commutations  of  readability.  The  Cloze 
techniouo  probably  accounts  for  comprehension  better  than  any  other 
available  reading  formula,  tut  the  Cloze  score  ray  be  confounded  vy 
vocabulary  development  and  readine  sneed.  'T'his  deficiency  most  litelv 
results  from  the  dependent  relationship  which  Cloze  scores  have  with 
standard  measures  of  readinq  shill  (reference  4) . The  validity  problems 
associated  v’ith  standard  tests  of  readinq  shill  further  confound 
interpretation  of  Cloze  scores. 

Cany  of  the  alove  criticisms  of  existino  readability  formulas, 
esrecinllv  the  Cloze  techniouo,  have  been  fade  previously  in  a report 
!'•  Carver  (reference  ")  . In  this  rejxsrt.  Carver  also  developed  a 
now  readability  formula,  the  Raudinq  Scale,  which  appears  to  correct 
many  of  the  deficiencies  associated  with  the  earlier  formulas  by 
acccur.tinq  for  a comprehension  factor.  Similar  to  the  Cl  ratinq  method 
used  in  the  present  analysis,  the  ^audinq  technioue  uses  personnel  wbo 
are  familiar  with,  readinq  shill  development  (experienced  teachers  or 
qrariuate  students  enrolled  in  readinq  training  courses) , and  who  have 
passed  a screenino  test  (the  P.aud inq  Scale  mual if ication  "'p?t)  , to  judoo 
the  readability  of  textual  material.  These  iudoes  appear  to  use  both  the 
input  difficulty  (sentence,  word,  and  syllable  complexity)  and  storaqo 
difficulty  (the  difficulty  in  orqanizinq  and  inteqratinq  the  input  factors) 
in  determininq  the  readability  of  textual  materials.  Although  the 
validity  of  an  independent  comprehension  factor  withir  the  taudinq  Scale 
remains  to  be  demonstrated , this  technique  nonetheless  warrants  serious 
attention  ty  writers  and  nanauer s of  military  trainine  materials,  '"he 
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application  of  Kaudinq  Scale  techniques  to  the  present  'rprr  data 
would  appear  to  he  an  excellent  means  of  va ] idati pe  and  ostabl  i shing 
the  uscfujness  of  the  icauding  Scale  in  reasurine  ti  e readability  of 
military  traininq  materials. 

Inasmuch  as  the  alove  results  indicate  t^at  error  rates  in 
:!RCC  courses  arc  not  related  to  the  readability  of  either  the  r^f' 
passages  or  tlie  study  questions,  hut  are  associated,  instead  with  the 
conceptual  level  r ecu i red  to  ansver  the  study  questions  corroctlv  fror 
the  information  which  is  provided  in  tlie  associated  rrr  passages,  a 
problem  arises  as  to  whether  or  rot  the  conceptual  levels  of  the  difficult 
study  questions  should  he  made  easier  in  order  to  reduce  the  error  rate, 
two  lines  of  evidence  indicate  tiiat  althougl  lovorino  the  corce’  tual 
level  nav  reduce  error  rates,  such  a procedure  nav  innair  overall 
training  effectiveness,  first,  if  the  assumption  is  valid  that  the 
structure  of  MRCCs  (especially  the  verhation  characteristic  of  nonerror-- 
low  CL--.^.udy  questions)  permits  students  to  read  onlv  a fraction  of 

the  Mis  (ref erenc^*  10)  , then  the  more  aneilytical  and  intoaratl ve 

* 

characteristics  of  the  hint,  CX.  study  questions  would  probably  result  in 
tore  thorough  reading  of  t'nc  UTI's  in  order  to  obtain  the  information 
necessary  to  answer  these  studv  questions  correctly.  Tie  effect  of  this 
more  extensive  readinq  should  be  improved  learning  of  the  textual 
material,  especially  improved  learninq  of  material  that  is  incidental 
to  answering  tlie  study  questions  correctly.  Secondly,  some  evidence 
(reference  14)  shows  that  textual  material  that  is  associated  with  high 
CL  test  items  is  remembered  better  (after  7 days)  than  textual  material 
that  is  related  to  low  CL  (verLatir.i)  test  items.  This  retention  effect 

20 


jL 


s ft  ur.d  to  he  most  sianificant  for  training  situations  ir  which  the 


hiyh  hi.  * est  iter  s were  administered  ir  a pre-test  fashion,  whir?  ir 
similar  to  the  n >’  mio'*  ion  tec  hr  i nnp  u^fi  jr  »*err«j . Pioh  r*r  ter.t 

items  (study  questions)  appear  to  result  in  hotter  structured  arid 
organised  lone-term  rc-norv  thar  low  Cl.  test  iter’s  (study  one st ions)  . 

Chose  data  demonstrate  that  while  high  Cl.  study  questions  ma”  he  pore 
difficult  to  learn  initially  than  low  Cl.  euert ior.s , tie  imnrovod 
retention  of  1 in':  CT.  cuestions  once  lcarrcd  nay  male  such  ouestionr 
more  trainino  effective  than  low  CL  questions,  "’he  use  of  high  CT. 

(high  error)  study  questions  in  LRCCs  should  continue  therefore  1 ecaus" 
these  t”oep  of  nuestions  appear  to  make  TJPCC  students  read  more  of  the 
nr  material  and  remember  the  material  which  thev  do  read  better  than 

if  these  nuestions  were  not  included  in  the  course  materials.  Perhaps 

r 

the  trair.i  no  effectiveness  of  THCCs  could  be  improved  substantial  1 v 
if  more  hiul  CL  study  questions  were  included  in  the  present  courses. 

The  order  or  distribution  of  hinl  and  low  CT.  studv  nuestions 
within  ar.  assignment  nay  also  have  important  training  consenuences . 
Although  each  of  the  present  lilCC  assignments  specifies  several 
instructional  objectives,  the  written  material  associated  with  these 
objectives  (studv  nuestions  and  PTKs)  do  not  appear  to  nronress  in  a 
systematic  fashion  from  simple  to  the  more  complex,  levels  within  the 
connitive  domain  developed  hv  Moor  (reference  12).  At  present,  these 
written  materials  appear  to  he  associated  randomly  with  each  objective. 
Perhans  f'RCCs  would  lie  more  training  effective  if  these  written  materials 
were  organized  initially  around,  basic  knowledge  factors  (as  indicated 
by  study  nuestions  with  low  CL  ratines)  and  progressed  systematically 
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to  the  more  complex  cognitive  factors  of  application  and  analysis 
(as  indicated  by  study  questions  with  high  CL  ratings) . This  method 
would  require  that  (a)  the  training  objective  be  stated  in  measurable 
terms,  (b)  the  concept  be  presented  in  a series  of  training  items, 
and  (c)  performance  (application,  analysis,  and  so  forth)  be  tested 
in  a terminal  item.  For  example: 


The  CL  rating  technique  described  in  this  report  could  be  used  to 
assist  NRCC  writers  in  rearranging  and  reorganizing  the  written  materials 
associated  with  instructional  objectives.  In  this  way,  NRCC  students 
may  be  better  prepared  to  understand  and  answer  correctly  the  more 
difficult  study  questions,  as  well  as  have  better  retention  of  both 
simple  and  complex  information  for  longer  periods  than  may  presently 
be  the  case.  The  improved  training  effectiveness  of  NRCCs  promised  by 
this  rating  technique,  as  well  as  the  ease  of  applying  the  technique  to 
present  NRCCs,  should  warrant  an  evaluation  of  the  technique  in  the  near 
furture . 
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The  rT  rating''  < escril  pd  i I'  tltir  reporf  > a"  ,il  71  i ,v'e  ipo  f 
aj  1 licat  ior.  in  tl  .<•  r a na<rrr  or t of  ‘"'.CCs.  In  review* the  rrrr 

the  error  -a!  pie,  a!  on*  *17 . e,  for  cent  of  t':o  study  euestiops  ir  t*  r 
e,T  "le  verr  *ound  to  l.avr  a Cl.  rat  inn  or  2 or  lor?;.  -Hr  finding 
i re'  i cat  on  that  these  study  euostior.s  rer-uirn'.  iittir  t!ir!  * r.n  fe 
arrw,  ’ ut  \:orr  nonetheless  frenuent  )"  missed.  ""ho  judges,  as  t oll 
as  the  authors  of  this  report , perceived  these  study  oueRtion*-  to  to 
roorly  vritten--airhiyuous,  misleading,  or  deceptive  ( 'tr  icl  v”  ) , or  cor- 
t a i nine  f.iultv  technical  irfornat.ion.  'sine  sarrlrr  of  rer.ilar  'ore 
students  in  the  future,  cor  nari  sons  could  t o nado  hrtv/oon  the  error 
rates  arc!  the  CL  ratines  for  study  ouestions  in  ar.  ’’PCC.  Those  study 
nueetiors  with  hint,  error  rates  and  loir  CT,  ratines  could  then  he 
identified  and  reworded,  repositioned,  or  eliminated.  7s  demonstrated 
in  this  report,  this  technioue  would  he  inexpensive  because  judnes  can 
hr  trained  easily  and,  with  little  exnerioncr , a iudoe  car  rate  reliable 
1C  to  20  study  ouestions  an  liour  . 

Another  application  of  this  ratine  technioue  would  ho  to  judge 
the  conceptual  difficulty  of  information  contained  in  nraohs  and  fiqures. 

7>t  present,  a rrlial  le  technioue  or  forrula  similar  to  those  used  to 
measure  the  readability  nf  textual  material  does  rot  exist  for  oraphir 
materials.  The  use  of  judges  and  a rating  scale  as  described  in  this 
report  could  provide  a measure  of  the  conceptual  processing  that  nav 
1 e necessary  to  understand  the  information  contained  in  oranhic  material, 
or  in  combinations  of  graphic  and  textual  material. 

"'ho  above  evidence  indicates  that  more  research  should  be  done  to 
1 earure  retention  of  dec  information  over  long  periods,  and  that  difference 
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in  intention  1 etveer  information  rortai mid.  in  high  and  low  Cl  stud*' 
cue rt ions  should.  I o innlvzed.  '"Mr.  rrsr.irc1  s'ou  Id  aim  document 
v/hctlcr  or  rot  .'ore  extensive  roaOinu  of  rib's  (an  d er on st rated  by 
incidental  lcarrir.g)  occur*'  in  association  - i th  hioh  f I.  tlan  lo”  Cb 
studv  cuestions.  An  analysis  should  bo  rude  ar.  '.-ell  of  the  trainin** 
cf f -ctiveness  (learning  anu  rrti'iition  of  studv  questions.)  resulting 
fro?  rearranging  and  reorganizing  (hut  not  rewriting)  the  written 
raterials  (FTMs  ana  study  cuestions)  associated  with  1 ’PCC  instructional 
obicctives.  Fuch  rearrangement  and  reorganisation  would  occur  in  the 
progressive,  systematic  (simple  to  complex)  fashion  described  above 
according  to  the  C.'.  ratines  which  hnvr  been  assigned  bv  iudner,  bo  the 
studv  cuestions  tl  at  are  used  to  sunnort  the  instructional  oh-jectivrs. 
Inasmuch  as  J’HCCs  are  a primary  training  medium  for  the  Pavy  Reserve, 
and  arc  also  an  imiortant  ingredient  in  the  advancement  of  Paw 
enlisted  personnel,  those  findings  would  have  implications  for  t>>g 
preparedness  of  the  Pavy  reserve,  as  well  as  the  va) iditv  and  management 
of  the  advancement  in  rate  process. 

H.n  COMCLL'fldUr . The  followinc  conclusions  arc  derived  fror  the  av'ove 
findings: 

8.1  ilP.CC  perforrarce,  as  represented  by  error  rate  on  study  cuestions, 
does  not  ap>pear  to  he  related  to  the  readahilitv  love]  of  iTCC  stud** 
questions  or  associated  Kli!  passages  as  determined  by  a popular, 
conventional  measure  of  readability  (the  rlesch  beading  Fase  Fcore)  or 
by  a readability  measure  v.’hich  lias  1 een  developed  speeificallv  for  ’*avv 
enlisted  personnel  (the  Kincaid  formula). 

8.2  "ho  extent  to  which  thinhinc  or  conceptual  processor  are  involved 
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in  answer  in<;  ti.c  stuuv  questions  which  is  referral  to  an  Ooncent.tial 
Level  (n.)  , i >■  related  significantly  *o  ' tin’''  ruestior  error  irate, 
o.  1 Concertual  i.ovelr  cai.  he  i.eten  iml  reiia!  ly  and  ouic’i  Iv  i v ■juc’.no'; 
vlio  .ire  i.i.  fan  j liar  with  ...*CC  content  or  devote  rnrt. 

C .A  Poor  1 v written  rater'll';  (either  rii^rt  ions  or  "sroc  later" 

IT  f a?:  sages)  r ay  l<  ident  < ' i od  rcaoily  as  those  materials  w!  ic1'  ar 
vrittei  at  a lov  < i.  1 ut  vhich  have  : i<d  error  rates. 

o.S  ' vali'.i  detomj nation  could  not  be  trade  or  the  e: tent  to  v->  ic’  (a) 
then-*  ;,RCC  materials  (for  the  \ri2u2  course)  were  representative  of 
other  ’Wr,  or  (1  ) the  answers  of  this  sample  of  I " s i r.  eereral 
v.ru  1<’.  ho  ty  ical  of  other  ..'.ivy  ratines.  '.  > ese  f i nd i nos  do  indicate 
that  1 Vs  appear  to  have  significantly  lower  GOT  scores  tl  an  similar 
rate  levels  ir  other  ratines,  ar.d  have  raert  fever  norths  in  rate 
I,  anu  fewer  vears  in  the  bavy  before  tal  ing  the  advancement  eiramr  for 

cor.tarailo  rate  levels. 

O.h  M CO: ."!  . bY.' Tf'dL . The  follov.ino  rceorinendations  are  made  in 
association  vith  the  alow  conclusions; 

°.l  "he  oresont  find.irK.s  should  !e  replicated  urine  different  "I’CCm, 
rates,  and  ratines.  In  addition,  the  <1-ijoint  Conceptual  I.evel  ratino 
scale  should  i.e  applied  Ly  different  judqcs  to  these  other  h'l  CC 
materials  in  order  to  determine  the  reliali.lit''  of  the  CL  ratine 
technique. 

0.2  heart a l il i ty  formulas  should  ’e  evaluated  w!  icn  include  sore  measure 
of  the  thir.V ing  rcouired  to  'mder stand  and  use  textual  material.  A 
rcatial.  ility  formula  vhich  should  ),o  evaluated  in  tliis  manner  is  the 
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Paudjno  Reals. 


9.3  7 determination  should  >e  made  of  the*  relationship  between  ver'nl 
intelligence  (us  measured  by  OCT)  a no  per formance  (error  rates)  on  hiah 
and  low  Cl-  study  questions.  fuch  a determination  would  pem.it  con- 
current validation  of  the  CT,  techniques,  as  well  as  trovi do  jpCorration 
aiiout  reading  and  other  I’P.cr  aptitudes  and  skills  which  could  perhaps 
he  inproved  through  education  and  training. 

9.4  lore  should  )e  done  to  analyze  lor.q-tern  retention  of  the  infor- 
mation presented  in  Jiigh  and  low  CT,  study  questions,  as  veil  as  the 
incidental  learning  which  nay  occur  in  association  with  answer ine  Mo! 
and  lov.<  CL  stvidy  questions.  'These  results  would  shov;  whether  more  or 
rewer  high  CT,  studv  questions  should  he  included  in  mpco. 

9.5  Tlie  training  effectiveness  of  orgarizino  study  questions  (ar.d 
associated  RTf’  material)  in  a nronressive,  systematic  fashion  from 
lov;  to  high  CL  ratings  in  relation  to  instructional  cl  iectivcs  s'  ould 
be  determined.  Such  an  organization  may  result  in  easier  learning  and 
Letter  retention  of  hiah  CL  material  than  would  be  true  for  tie  present 
tmCCs  in  which  much  of  the  raterial  is  presented  unsystematically. 

S.G  Research  should  he  conducted  into  using  the  Cl,  ratine'  technique 
(or  a similar  technique)  for  iudqing  the  conceptual  difficulty  or 
comprehensibility  of  graphic  material.  The  information  resultino  fror 
this  research  would  indicate  the  aptitude  and  skill  level  (and 
associated  training)  required  to  understand  and  use  graphic  materials. 
9.7  Conceptual  Level  iudgmerts  should  be  used  to  correct  or  discard 
faulty,  ambiguous,  misleading,  or  deceptive  ("tricky")  studv  questions 
and  TOT  j-tassages  fror  KRCCs.  by  idertifvinc  this  ratcria)  th.rouqh  lov 


CT  study  eurrtions  which  arc  associated  wi*v-  >- j «i»>  r-rrnr  rates,  rorc 


objective  am4  irprovcvl  oual ity  control  would  >e  exerej'-o^  over  ’’err 
drve] orrer.t , and  better  trainino  should  result. 
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11.0  FOOTNOTE? 


1.  Lieutenant  Commander  Biersner  was  Programs/fjystems  Effectiveness 
Evaluation  Analyst  (Code  01A2)  at  the  Naval  Education  and  Training 
Support  Command,  and  is  now  Research  Projects  Officer  (Code  01), 
Naval  Submarine  Medical  Research  Laboratory,  Naval  Submarine  Base 
New  London,  Groton,  Connecticut  06340.  Mr.  Doucette  is  a 
Mathematical  Statistician  at  the  Naval  Education  and  Training 
Program  Development  Center,  Tensacola,  Florida  32509. 

2 The  subject  group  varied  by  plus  or  minus  two  subjects  from  this 
figure  (N  = 30)  denending  on  whether  or  not  a subject  failed  to 
submit  one  or  more  of  the  twelve  assignments  described  below  in 
section  5.2. 

3.  The  characteristics  of  a well  written  NRCC  are  (a)  the  written 
material  is  presented  in  a grammatically  correct,  organized,  and 
concise  fashion,  (b)  the  information  is  technically  accurate, 
and  ( c)  the  study  (questions  are  judged  to  be  fair  and  valid 
(face  validity) . 

4.  Section  6 1 provides  evidence  that  this  judgement  was  not  entirelv 
■'alid. 

5.  A Kincaid  PGL  was  not  computed  for  the  HTM  3 & 2 assignment 
booklet  because  the  Kincaid  formula  was  developed  only  from 
sample  passages  selected  from  RTMs  (see  reference  1 for  details) . 

6.  These  parameters  were  (a)  less  than  five  or  more  than  37  words 
per  sentence,  (b)  less  than  120  or  more  than  200  syllables  per 
100  words,  and  (c)  a Flesch  Reading  Ease  Score  of  less  than  0 
or  more  than  100. 
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Svr-».-y  Statures  cr  • rb.1  te’Mrance  ar.d  Carear-Dita 
Navy  Dcrsor.re1  T a tt ' *■  9 • ar.- .rvant  in  Rate  Examinations 


Sirr.ary  Statistics  cf  Flasch  ar  i \incaid  Readinq  G-ada  Lewis  (RGLs) 
for  RTF'  Passages  aid  Study  Questions  in  the  Error  ana  N'onerrcr  Samples 


APPENDIX  A 

The  4-Poirt  Ration  Scale  Used  to  Judoe  the 
Conceptual  Level  (CL)  of  Study  Questions 
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CONCEPTUAL  LEVEL  1 (CL  = 1) 


The  answer  and/or  question  stem  are  taken  verbatim  or  nearly 
verbatim  from  the  manual.  Even  if  the  meaning  of  some  of  the  words  is 
not  totally  understood,  the  answer  is  obvious  because  the  words  and 
sentences  in  the  study  question  nearly  duplicate  the  words  and  sen- 
tences in  the  manual.  Included  under  this  rating  would  be  those  cases 
in  which  some  of  the  words  from  the  manual  may  have  been  removed  from 
the  study  question  (or  v: ce  versa) , but  the  key  words  remain  in  the 
same  order  (rearrangement  of  key  words  is  therefore  unnecessary) . 

CONCEPTUAL  LEVEL  2 (CL  = 2) 

Simple  reordering  or  rearrangement  of  some  of  the  words  and 
sentences  in  the  manual  is  necessary  to  answer  the  study  question.  The 
answer  and/or  question  stem  paraphrases  the  wording  in  the  manual.  For 
example,  numbers  presented  in  graphic  form  in  the  manual  are  translated 
or  converted  to  verbal  form  in  the  study  question  (or  vice  versa) , or  a 
positive  statement  in  the  manual  is  reversed  to  a negative  statement  in 
the  study  question  (or  vice  versa) . Most  of  the  thinking  is  involved  in 
rewording,  reordering,  or  reversing  the  sentences  and  words  in  the  manual, - 
after  this  process  is  complete,  little  thinking  is  required  to  answer  the 
study  question. 

CONCEPTUAL  LEVEL  3 (CL  = 3) 


A simple  situation  or  problem  described  in  the  study  question 
requires  the  application  of  one  or  more  general  procedures  or  concepts 
presented  in  the  manual.  All  of  the  information  necessary  to  answer 
the  study  question  is  provided  in  the  manual  or  the  stem  of  the  question, 
and  the  interrelationships  between  steps  in  the  procedures  or  between 
concepts  have  been  fully  specified.  Some  care,  however,  may  still  be 
required  in  following  or  applying  these  procedures  or  concepts.  For 
example,  the  answers  may  be  similar,  and  may  vary  only  slightly  according 
to  minor  deviations  in  following  or  applying  the  procedures  or  concepts 
described  in  the  manual;  or  the  information  required  to  answer  the  study 
question  may  be  located  in  several  widely  separated  sentences  in  the 
manual,  and  the  reader  must  organize  the  sentences  together  in  order  to 
answer  the  study  question.  A high  level  of  care  and  discrimination  is 
therefore  required  in  answering  the  study  question.  Although  the 
procedures  may  be  straightforward,  some  thinking  is  required  in  order 
to  insure  that  the  information  or  conditions  which  are  specified  in  the 
study  question  are  applied  in  the  exact  sequence  or  fashion  that  is 
indicated  in  the  manual.  Moderate  levels  of  thinking  are  required  in 
converting,  translating,  and  organizing  the  specific  conditions  or 
information  provided  in  question  stem  and  answer  to  the  general  procedure 
or  concept  presented  in  the  manual.  The  reader  is  usually  certain  of  the 
correct  answer  after  rechecking  the  steps  or  log;  that  have  been  prescribed 
in  the  manual. 
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CONCELTUAL  LEVEL  4 (CL  = 4) 


Answering  this  type  of  study  question  also  requires  the  application 
of  one  or  more  general  concepts  or  procedures  which  are  presented  in  the 
manual.  Some  of  the  relationships  between  the  concepts  or  steps  of  a 
procedure  are,  however,  either  vaguely  described  or  remain  unspecified 
in  the  manual.  Much  of  the  interpretation,  integration,  and  organiza- 
tion of  the  concepts  or  procedures  provided  in  the  manual  must  therefore 
be  done  by  the  reader  in  order  to  understand  and  answer  the  study  ques- 
tion. The  reader  may  also  have  to  make  assumptions  about  the  relationship 
between  information  provided  in  the  manual  and  information  provided  in 
the  study  questions.  For  example,  the  study  question  does  not  present 
all  of  the  information  necessary  to  determine  which  of  the  answers  most 
fully  meets  the  conditions  presented  in  the  manual  or,  vice  versa , the 
study  question  contains  information  or  conditions  which  are  not  specified 
in  the  manual.  The  reader  is  often  uncertain  of  the  correct  answer 
because  of  the  assumptions  that  must  be  made  in  interpreting  concepts 
and  procedures  in  the  manual,  or  in  applying  these  concepts  and  procedures 
to  the  conditions  specified  in  the  study  questions. 
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APPFI’nTX  P 

rxanples  of  rtudy  cuections  and  Associated  p'**' 
Passages  luited  at  lach  of  the  4 Conceptual 
t.evels  (Cl.r)  by  the  ? Judqes 


3f 


CONCEPTUAL  LEVEL  1 (CL  = 1) 


Study  Question: 

The  standard  taper  of  pipe  threads  per  foot  is 

1.  1/4  in. 

2.  1/2  in. 

3.  3/4  in. 

4.  1 in. 

Associated  KTM  Passage: 

The  NPT , which  formerly  stood  for  National  Pipe  Thread,  is 
still  used  as  a carryover  and  now  refers  to  the  new  name  for  the 
same  thread,  American  Standard  Taper  Pipe  Threads.  The  standard 
taper  of  pipe  threads  is  3/4  inch  per  foot.  The  number  of  threads 
per  inch  varies  according  to  the  size  of  pipe  as  follows:  1/16- 
and  1/8-inch  pipe  have  27  threads  per  inch;  1/4-  and  3/8  inch  pipe 
have  18  threads  per  inch;  1/2-  and  3/4-inch  pipe  have  14  threads  per 
inch;  1-,  1 1/4- , 1/2- , and  2-inch  pipe  have  11  1/2  threads  per  inch; 
2 1/2- inch  pipe  and  pipe  larger  than  2 1/2  inches  have  8 threads  per 
inch. 


CONCEPTUAL  LEVEL  2 (CL  = 2) 


Study  Question: 


The  gaskets  used  between  a grinding  wheel  and  it  retaining 
washers  should  be  made  of 

1.  iron  or  steel 

2.  fiber  or  rubber 

3.  paper  or  cardboard 

4.  copper  or  brass 

Associated  RTM  Passage: 

The  following  operating  instructions  and  safety  precautions  are 
applicable  in  general  to  all  grinders  and  specifically  to  the  bench 
and  pedestal  types. 


1.  When  starting  a grinder,  push  the  start  button  and  stand  to 
one  side  for  at  least  one  minute  while  the  machine  comes  up  to  full 
speed.  There  is  always  a possibility  that  a wheel  may  fly  to  pieces 
when  coming  up  to  speed. 

2.  Never  force  work  against  a cold  wheel;  apply  work  gradually  to 
give  the  wheel  an  opportunity  to  warm.  You  thereby  minimize  the  possi- 
bility of  breakage. 

3.  Handle  wheels  carefully.  Before  replacing  a wheel  on  a grinder, 
always  sound  the  new  wheel  for  cracks.  To  sound  a wheel,  tap  it  lightly 
with  a piece  of  hard  wood;  a good  wheel  gives  oi t a clear  ringing  sound 
when  tapped,  but  if  the  wheel  is  cracked,  a dul ’ tnud  is  heard.  Make 
sure  that  a fiber  or  rubber  gasket  is  in  place  between  • ach  side  of  the 
wheel  and  its  retaining  washer  (spindle  wheel  flange) . Tighten  the 
spindle  nut  just  enough  to  hold  the  wheel  firmly;  if  the  nut  is  tightened 
too  much  the  clamping  strain  may  damage  the  wheel. 
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GOWCfPTAL  I.H'TT  "1  (CL  = 3) 


f> t uc.y  I Me stji > n : 

After  the  frames  arc  positioned  apti  first  secured  to  the  keel, 
tk.ev  tire  bolted  or  riveted  to  clamp  them  between  the 

1.  keelson  and  I eel 

2.  ribbands  and  keel 

3.  planking  and  keel 

4.  carloard  strake  and  keel 

Associated  1 ."’T  ras^aae : 

'.'her.  a 5 oat  shop  is  ordered  to  build  a boat  the  following  nlars 
arc  furnisl.ee:  lines  and  off-sets,  inhoard  and  outhoard  profiles, 

am id shin  sections,  and  a data  sheet  which  gives  the  scantlin^  or  sizes 
of  the  principal  members  of  the  frarine.  With  this  information,  the 
>oatbuilders  proceed  to  cut  out  and  s^ane  the  various  narts  of  the  hoat. 

the  first  step,  in  assembling  a boat  is  to  place  the  ^eel  on  blocks 
which  are  heavv  enough  to  support  the  weight  of  the  loat  and  high  enough 
to  allow  for  working  room.  Then  the  stem  and  sternpost  are  riveted  or 
lolled  te  the  heel  and  held  riqidlv  in  position  bv  temper arv  shores  and 
braces . 

'ext,  the  ends  of  each  pair  of  frames  are  butted  and  secured  hir 
thwartshi;  timbers  known  as  FLOORS.  Then  the  frames  are  put  in  position 
and  secured  to  the  keel.  Filling  pieces  arc  then  fitted  over  the  keel 
and  between  the  frames.  Fext  the  keelson  is  placed  over  the  frames, 
cloorr,  and  filling  pieces  and  directly  above  the  'eel.  bolts  or  rivets 
are  used  through  the  keelson,  filling  pieces,  and  keel,  to  clamp  and  w0ld 
the  frames  securely  in  place.  The  portion  or  end  of  the  frame  set  to  keel 
is  known  as  heel.  The  other  end  to  top  sides  is  known  as  head.  After 
frames  are  posit ioned  and  secured  to  the  keel,  they  are  lined  up  in  the 
vertical  direction  and  secured  with  temporary  wooden  batters  of  beavv 
sections  to  fix  them  rioiely  in  final  position,  '"here  witters,  'r.own 
to  ti  c bcatkuilder  as  rib  -l  ands,  are  installed  armroxinately  horizontal. 
Any  inside  horizontal  framing  r°auired  in  the  hull  nav  be  installed  on 
forms  prior  to  setting  transverse  framing,  or  after  frames  are  removed, 
lull  design  dictates  which  method  should  be  followed. 
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' ti.'  v ' uestio:  : 

Fav’ne  iust  replaced  a depleted  cylinder  of  oxygen  ir:  a mortahle 
oxyacetylene  outfit,  you  reconnect  tl  «»  hose  an.  norcv  . 'bat  are  the 
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Associate*:  Passant: 

r>.  i /Or'.rt: : A’U*  * :a:t:  act : triors  the  s-.vnt.er  i*  tliorougi  ]«  purged 

of  air  an:  all  connections  it'  the  svst»r’  arn  tiaiit  ) ofore  the  torch  ir 
ionite*.,  the  flare  is  likely  to  Lurn  inside  the  torch  instead  of  outside 
the  ti*  . The  difference  between  the  two  terms  backfire  and  f 1 arhhae)  is 
this:  in  a lac)  fire,  there  is  a momentary  ’ -urn  inn  ) ac)  of  the  flare  into 

the  torch  ti:  ; it  a flashback,  the  f lane  turns  in  or  Levon'  the  torch 
r.ixirg  charier.  * 1. tel. fire  ic  characterised  hy  a loud  man  or  mo  as 
the  flare  goes  out.  A f lashl-ac!;  is  usually  accomnaried  -s>v  a ) issing  or 
s-  ucalinc  sound.  At  the  sane  tire,  the  flare  at  the  tin  becomes  sro'  v 
ot  i sharp  ;ointed . ’.Then  a flashi/acl  occurs,  immediately  shut  off  the 
rcl  oxygen  valve,  then  close  the  acetvler.e  valve,  "'he  occurrence  of 
■i  flar.htac'  indicates  that  someth  i rn  is  radical!''  wrong  either  *m  th 
the  torch  or  with  tie  manner  of  hand  linn  it.  A.  lachfire  in  lens  serious. 
Usually  the  flane  can  he  rcl  iohtec  without  difficult'/.  Tf  bachfirina 
continues  w)ierevf:r  the  torch  is  relighted  checl.  for  these  causes:  over- 

heated tit-,  qas  vnrhinq  pressures  area ter  than  that  recommended  for  the 
tir  sice  feint:  umi,  loose  tin,  or  dirt  on  the  torch  tin  seat,  '"hose 
same  dif f iculties  may  be  the  cause  of  a fit shbacl , except  that  the 
difficulty  is  present  to  a greater  degree.  Tor  eyamnle , tie  tore*"  heat’ 
mav  he  distorted  or  crach.eu. 

In  most  instances,  had. fires  and  flashbacks  result  from  careless- 
ness. These  difficulties  can  be  avoided  bv  na!  ino  certain  that:  (]) 

all  connections  in  the  svster  are  clear,  and  tia’  t;  (?)  torc^  valves 
are  closed  (not  onon  or  merely  loose lv  closet))  v^en  the  ecuinmrr.t  is 
stoweu;  (3)  the  oxygen  ant:  acetvlcne  working  fressures  use<'  are  those 
recommenced  for  the  torch  employed;  and  (4)  the  system  is  r urged  of  air 
before  the  ajraratus  is  used . Purging  the  system,  of  air  is  es'>ecial]lv 
necessary  when  hose  and  tore!’  ''ave  been  nowl"  ernneeted  or  a sew  cvlirder 
is  incorjiorated  into  the  syster . Purging  a system  is  accomplished  as 
follows:  Close  tore!',  valves  tightly,  then  slowlv  oner  the  cylinder 

valves.  Next,  open  the  regulator  slightly.  t per.  the  torch  acetylene 
valve  and  allow  acetylene  to  escape  for  r-  to  If  seconds,  depend  ino  on 
the  length  of  the  l.ose.  Close  the  acetylene  valve.  Repeat  the  pro- 
cedure or  the  oxyger  side  of  the  system.  After  purqiro  air  from  the 
system,  light  the  torch  as  described  previously. 
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